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Abstract 

When testing for replication of results from a primary study with two-sided 
hypotheses in a follow-up study, we are usually interested in discovering the 
features with discoveries in the same direction in the two studies. The direction 
of testing in the follow-up study for each feature can therefore be decided by 
the primary study. We prove that in this case the methods suggested in [lj for 
control over false replicability claims are valid. Specifically, we prove that if we 
input into the procedures in [1] the one-sided p -values in the directions favoured 
by the primary study, then we achieve directional control over the desired error 
measure (family-wise error rate or false discovery rate). 


1 Introduction 


In this note we are concerned with the setting where two studies (a primary study 
and a follow-up study) are available that examine the same problem, and the aim 
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is to discover the features with true findings with directional consistency from the 
primary to the follow-up study. So features with true left-sided alternatives in both 
studies are of interest, and features with true right-sided alternatives in both studies 
are of interest, but a feature with a true left-sided alternative in one of the studies 
and a true right-sided alternative in the other study is not. 

In [lj we suggested procedures for declaring that findings from a primary study have 
been replicated in a follow-up study, where a feature has a false replicability claim if 
the null hypothesis is true in at least one of the studies. Our proposal assigned an 
r-value to each finding. The false discovery rate (FDR) r-value for feature i is the 
lowest FDR level at which we can say that the finding is among the replicated ones. 
We showed that the the procedure that declares findings with FDR r-values below 
q as replicated controls the FDR on replicability claims. For family-wise error-rate 
(FWER) control on replicability claims, we suggested using FWER r-values with a 
similar property. In this note, we extend the results in [1] for the setting that the 
direction of the effects is unknown in advance, and the direction of testing is de¬ 
termined by the data of the primary study. We suggest using for each feature the 
minimum one-sided primary study p- value, i.e., we consider for replicability the di¬ 
rection the data favours, and in the follow-up the one-sided p-value in the favoured 
direction determined by the primary study. For example, if in a primary genome¬ 
wide association study (GWAS) the direction of association is unknown in advance, 
the primary study serves to guide two important design decisions for follow-up: first, 
which hypotheses will be followed-up, and second, the direction of testing in the 
follow-up study. For replicability analysis, we will need the one-sided p -values in the 
primary and follow-up studies, in the direction of association determined by the pri¬ 
mary study. Although we decide for each feature the direction of testing based on 
its test statistic in the primary study, if we compute the r-values as in [1], then the 
procedures that declare as replicated all features with all r-values below the nominal 
level control the directional FDR/FWER. This is remarkable, since we are used to 
paying a factor of two for a single study when the direction of testing is unknown: 
we multiply the (minimal) one-sided p-value for two-sided hypothesis testing. For 
replicability analysis, there is no cost of not knowing before looking at the results 
from the primary study the direction of testing for replicability. The reason is that 
these procedures already have a cost for the fact that we select the promising hy¬ 
potheses from the primary to the follow-up, and this cost actually covers also the 
selection of direction of interest. The r-values can be computed using our web ap¬ 
plication http://www.math. tau. ac . il/$\sira$ruheller/App.htral. An R script is 
also available in RunMyCode,http://www. runmycode.org/companion/view/542. 


2 Notation 


Here we give the formal framework for replicability analysis for two-sided hypotheses, 
including directional errors. Consider a family of m > 1 features examined in the 
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primary study. Feature j G {l,...,m} in study i G {1,2} has either a true or a 
false null hypothesis. If the null hypothesis is false, the true left-sided or right-sided 
alternative is true. We define H^ as follows: 


Hij 


1 if the right-sided alternative is true for feature j in study i, 
0 if the null hypothesis is true for feature j in study i, 

—1 if the left-sided alternative is true for feature j in study i. 


Let H — {h — (h\, h 2 ) : hi G { — 1, 0,1}} be the set of 9 possible configurations of the 
vector Hj = (Hij, H 2j ) for two-sided alternatives. (If interest lies only in detecting 
left or right sided alternatives, only the relevant subset of 4 possible configurations are 
considered.) The set of m features can be divided into 9 (unknown) subsets defined by 
H : {j : Hj — h},h G H. Each feature j is in exactly one of the 9 subsets. The subsets 
of features whose effect is not replicated in the same direction in the two studies are 
O' : Hj = h}, for h G H 0 = {(-1,1), (-1, 0), (1, 0), (0, 0), (0, 1), (0. - 1), (1,-1)}. 
The goal in inference is to discover as many features as possible with Hj ^ H°, i.e., 
HjE {(1,1), (-1, -1)}. 

Suppose R replicability claims are made by an analysis. Let R T - and R? be the 
indicators of whether a replicability claim was made for feature j in the left and right 
direction, respectively. The number of replicability claims that are true (i.e., that 
are with true left-sided alternatives in both studies or true right-sided alternatives in 
both studies) is 

E R f+ E R }’ 

{j:Hj=( 1 , 1 )} {r-Hj=(- 1 ,- 1 )} 

and R — S is the number of replicability claims that are false (i.e., that are with true 
left-sided alternatives in at most one study and true right-sided alternatives in at 
most one study). 

The directional FWER criterion is the probability of at least one false directional 
replicability claim, 

FWER = Pr(R- S >0). 

The directional FDR for replicability analysis is the expected proportion of false 
directional replicability claims among all those called replicated: 

FDR = E ( R ~ S \ . 

\ma x(it, 1) J 


For feature j tested in the follow-up study, the left- and right- sided p -values for study 
i G {1,2} are denoted by pL and pf-, respectively. For continuous test statistics, 
pfj = 1 — Pij. For replicability analysis, we will need the one-sided p -values only in 
the direction favoured by the primary study, i.e. the pair (p'ij,p' 2 j) defined as follows: 


Pii 


Pij if Pij < Pip 
Pij if Pij > Pij- 


P2j 


P2j if Pij < Pij, 
P2j if Pij > Pij- 
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Remark 2.1 We implicitly assume that p^ - < 0.5. This assumption may be violated 
if the test statistic is discrete. Since features withp[j > 0.5 are obviously not interest¬ 
ing features in the primary study, we suggest including in our selection rule (of which 
features to follow-up on) the condition that a feature is selected only if p\j < 0.5. 


3 FDR Replicability from follow-up for two-sided 
hypotheses 


As in P, we let foo denote the fraction of features, out of the m features examined 
in the primary study, that are null in both studies. We cannot estimate foo from the 
data, since only a handful of promising features are followed up in practice. However, 
foo is typically closer to one than to zero, and we can give a conservative guess for a 
lower bound on / 00 , call it Z 00 . For example, in typical GWAS on the whole genome, 
loo = 0.8 is conservative. We can exploit the fact that loo > 0 to gain power. 

3.1 Computation of r-values for FDR-replicability for two- 
sided hypotheses 

For completeness, we present the procedure for establishing replicability from follow¬ 
up, which is identical to the procedure in p. The only difference is that the one-sided 
p-values to input into the procedure are the ones favoured by the primary study. 

1. Data input: 

(a) m, the number of features examined in the primary study. 

(b) IZi, the set of features selected for follow-up based on primary study results. 
Let R\ = |72.i | be their number. 

( c ) { ip\j , P'-ijj) '■ j e 72i }, where p\ ] and p' 2 j are, respectively, the primary 
and follow-up study one-sided p-values for feature j G 72 1 in the direction 
favoured by the primary study. 

2. Parameters input: 

(a) loo G [0,1), the lower bound on / 00 (see above), default value for whole 
genome GWAS is Z 00 = 0.8. 

(b) C 2 G (0,1), the emphasis given to the follow-up study (see Section Varia¬ 
tions in HD, default value is C 2 = 0.5. 

3. Definition of the functions fi(x),i G 72i,x G (0,1): 

(a) Compute Cl (x) = 
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(b) For every feature j G compute the following e-values: 


( 1 / R i / A - ^ 

ej{x) = max —r~Piy - Pij , 

\Ci(a;) J mc 2 J 

(c) Let fi(x ) = min { j :£j - ( x )>ei(x) ,jcKi } r 2k[ej(x )] » where rarafcfejta;)] is the rank of 
the e-value for feature j G 7\li (with maximum rank for ties). 

4. The FDR r -value for feature i G IZi is the solution to /,;(r*) = ?y if a solution 
exists in (0,1), and 1 otherwise. The solution is unique, see Supplementary 
Information (SI) Lemma Sl.l in [I] for a proof. 

3.2 The level q directional FDR replicability procedure 

1. Compute the r -values as detailed in Section I3TT1 

2. The replicability claims at a prefixed level q, say q = 0.05, are all features with 
r-values at most 0.05. Denote this set of features by 1Z 2 . 

3. If feature j G 7 Z 2 has p\ rj = pf-, then declare the feature as having a replicated 
true left-sided alternative (i.e., a true effect/signal/association in the left direc¬ 
tion) ; If feature j G 1Z 2 has p\ ] = pfj, then declare the feature as having a 
replicated true right-sided alternative (i.e., a true effect/signal/association in 
the right direction). 


The directional FDR for replicability analysis is then controlled at level 0.05, as we 
show in Theorem 13.11 as long as the selection rule is stable. 

Definition JT]. A stable selection rule satisfies the following condition: for any 
j G TZ\, changing pf - so that j is still selected while all other primary study p-values 
are held fixed, will not change the set IZi. 

Stable selection rules include selecting the hypotheses with two-sided primary p-values 
below a certain cut-off, or by a non-adaptive multiple testing procedure on the primary 
study two-sided p-values such as the BH procedure for FDR control or the Bonferroni 
procedure for FWER control, or selecting the k hypotheses with the smallest two- 
sided p-values, where k is fixed in advance. 


Theorem 3.1 A procedure that declares findings with FDR r-values at most q as 
replicated controls the directional FDR for replicability analysis at level at most q if 
the following conditions are satisfied: the rule by which the set TZi is selected is a stable 
selection rule; Zoo < /oo>' the p-values within the follow-up study are jointly independent 
or are positive regression dependent on the subset of p-values corresponding to true null 
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hypotheses (property PRD^_ \); for features with Hj ^ {(1,1), (—1, —1)} the follow-up 
study p-values are independent of the primary study p-values; and in addition one of 
items 1-3 below is satisfied. 

1. The p-values within the primary study are independent. 

2. Arbitrary dependence among the p-values within the primary study, when in 
Step 3 in Section \3J\ m is replaced by m* = rn yj™, V*- 

3. Arbitrary dependence among the p-values within the primary study, and the 
selection rule is such that the primary study p-values of the features that are 
selected for follow-up are at most a fixed threshold t e (0,1), when c\ computed 
in Step 3(a) is replaced by 

\tm / (ax) — 1] 

C\(x) = max{a : a(l + S V*) = ci(z)}. 

i =1 

Steps 3(b) and 3(c) remain unchanged. In step f, the FDR r-value for feature 
i G IZ\ is ri = min{x : fi(x) < x} if a solution exists in (0,1), and one 
otherwise. 


See Appendix [A] for a proof. The implication of item 3 is that for FDR-replicability 
at level q, if t < ci(q)q/m, no modification is required, so the procedure that declares 
as replicated all features with r-values at most q controls the FDR at level q on 
replicability claims for any type of dependency in the primary study. Note that the 


modification in item 3 will lead to more discoveries than the modification in item 2 
only if t < v , ■ 


We conjecture from empirical investigations that even if the primary study p-values 
are not independent, the conservative modifications of the r- value computation in 
items 2-3 are unnecessary for FDR control in replicability analysis of GWAS studies, 
see [T] for details. 


4 FWER Replicability from follow-up for two-sided 
hypotheses 

4.1 Computation of r-values for FWER-replicability for two- 
sided hypotheses 

The directional FWER criterion, 

FWER = Pr(A — S > 0), 

1 Property PRDS was introduced in [3], For example, the PRDS property is satisfied if the test 
statistics are Gaussian, non-negatively correlated, and the tested hypotheses are one-sided. 
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is more stringent than the directional FDR, yet it may sometimes be desired. The 
directional FWER r-value is the lowest directional FWER level at which we can say 
that the finding has been replicated. The r-value can be compared to any desired 
level of directional FWER. For feature j G 1Z \, 

where ej(x ) is the e-value defined in Step 3(b) of Section I3T1 The Bonferroni r-value 
for feature j is the solution to ff° n A r j) = rj if a solution exists in [0,1), and one 
otherwise, ft can be shown that the solution is unique similarly to the case with FDR 
r-values. 


4.2 The level a directional FWER-replicability procedure 

1. Compute the r-values as detailed in Section |4~T1 

2. The replicability claims at a prefixed level a, say a = 0.05, are all features with 
r-values at most a. Denote this set of features by 7 Z 2 . 

3. If feature j G 7 Z 2 has pf- < pf), then declare the feature as having a replicated 
true left-sided alternative; If feature j G 7 Z 2 has pfj < pfj-, then declare the 
feature as having a replicated true right-sided alternative. 


The directional FWER for replicability analysis is then controlled at level 0.05, as 
stated in the following theorem. 

Theorem 4.1 The procedure above controls the directional FWER for replicability 
analysis at level a if loo < foo, and if for features with Hj ^ {(1,1), (—1, —1)} the 
follow-up study p-values are independent of the primary study p-values. 

See Appendix [B] for the proof. 
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A 


Proof of Theorem 


EL 


We first show that the following procedure is identical to that of declaring the set of 
findings with FDR-replicability r-values at most q as replicated. First, compute the 
number of replicability claims at level q as follows: 


Ro = max < r : 


je Th 


(Pijhlhj) < ( ^Ci(g)g, ^~C2q 


= r 


Next, declare as replicated findings the set 

^2 = |j : (p[j,P 2 j) < (q)q, |r c ? q ) , / e 77i| . 

It was shown in Lemma Sl.l in [1] that when the hypotheses are one-sided, this 
procedure is identical to declaring the set of findings with FDR-replicability r-values 
at most q as replicated, when the one-sided p-values replace (p']j,p 2 j) both in the above 
procedure and in the computation of FDR-replicability r-values. It is straightforward 
to see that the proof of Lemma Sl.l in [T] remains unchanged when the one-sided 
p- values are replaced by (p\j,p' 2 j), therefore the above procedure is identical to that 
of declaring the set of findings with FDR-replicability r-values at most q as replicated 
for two-sided hypotheses. 


We shall prove that under the conditions of items 1-3 of Theorem 13.11 the above 
procedure controls the FDR for replicability analysis at a level which is smaller or 
equal to 

Ci(g)c 2 g 2 (|j : Hj e {(-1, 0), (1, 0), (0, 0)}|)/m + 

ci{q)q\j ■ Hj G {(0,1), (0,-1), (-1,-1), (1,1), (-1,1), (l,-l)}|/m + 

c 2 qE[\n 1 n {j : Hj G {(-1, 0), (1, 0), (-1,1), (1, -1), (0,1), (0, -1), (0, 0)}|/|ft 1 |], 

( 1 ) 
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where the cardinalities are over the sets containing all m features, i.e. j = 1 ,,m. 
Note that this expression is at most q if l 0 o < foo- To see this, note that 

I j ■ Hj G {(-1,0), (1,0), (0, 0)}|/m = /. o, 

and 

I j : e {(0,1), (0, -1), (-1, -1), (1,1), (-1,1), (1, -1)}| /m = 1 - u 

Moreover, 

E[\K! n {j : E 3 G {(-1, 0), (1, 0), (-1,1), (1, -1), (0,1), (0, -1), (0, 0)11/1^1] < 1. 

Therefore, expression (JI]) is at most 

Ci(q)c 2 q 2 f-o + Ci(q)q(l - f. 0 ) + c 2 q 
= Ci(q)q - f.oCi(q)q(l - c 2 q) + c 2 q 
< C\{q)q - l 00 c 1 (q)q(l - c 2 q) + c 2 q 
= Ci(q)q[l - Zoo(l - c 2 q)\ + c 2 q 
= (1 - c 2 )q + c 2 q = q. 


We shall now prove that the expression in (JTJ) is an upper bound for the directional 
FDR for replicability analysis, which is 


( R - s ) 
\max(i?, 1) / 


E 

{j:^G{(0,-l), (0,1), (0,0), (1,0), (-1,0), (1,-1), (-1,1)}} 


max(R, 1) J 


+ 



( 2 ) 


For each j G {1,..., m}, we dehne cl ]} as the event in which if j is declared replicated, 
r hypotheses are declared replicated including j, which amounts to the definition given 
in the proof of Theorem 1 in Supplementary Material of [[[], where the one-sided p- 
values (pij,p 2 j) are replaced by (p[j,p 2 j)- Note that for any given realization of \1Zi\ 
and value of r such that r > |7£i|, C^ ] = 0. 


From the equivalent procedure above the following equality follows. 


m 


max(i?, 1) 


r —1 
m 


^-Pr ( j G 1Z \, Pjj < 


mm 


rci(q)q 


, 0.5 ) , P 2j < 


rc 2 q 


<ETN< rci(,k 


r =1 


m 


max(|7£i|, 1) 
,P£<c 2 q,Cp), (3) 


nti) 
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where the equality follows from the fact that a replicability claim is made in the left 
direction only if Py < Py, i.e. only if Py < 0.5. Similarly, 


E 


R 


R 


max(P, 1) 


< -Pr 

/ J p 

r —1 


?5< 


rci(q)q 


m 


Py < C2?,C«> 


(4) 


Using inequalities (J3]) and (J4]) , and the facts that Py and Py are uniform for j G 
{j : Hy = 0} and are stochastically larger than uniform for j G {j : Hy = 1} and 
j G {j : Hy = — 1} respectively, we obtain the following inequalities: 


/ r l \ ( ci(q)q/m if H, G {(0,-1), (1, -1), (1,1)}, 

E /o n < { c 2qE[I(j G P 1 )/|P 1 |] if Hj G {(—1, 0), (0,1), (—1,1)}, 
V 1 ( C^qfaf/m if Hj G {(0,0), (1,0)}, 


/ R R \ ( Ci(q)q/m if Hj G {(0,1), (-1,1), (—1,-1)}, 

E ’ <{ c 2 qE[I{jen 1 )/ |Pi|] if P,. G {(1,0), (0,-1), (1,-1), (0,0)}, 

1)/ 1 c l(9 )c 29 2 /m if/?,. 6 {(-1,0)}. 

These upper bounds for items 1-3 of Theorem 13.11 follow from similar derivations to 
these given in the proof of items (i)-(m) of Theorem 1 in [T], respectively. Specifi¬ 
cally, for each of the items, the upper bounds Ci(q)q/m, C 2 qE[I(j G Pi)/|Pi|] and 
Ci(q)c 2 q 2 / m are derived similarly to inequalities [S3], [S4], and [S5] in the proof of 
Theorem 1 in |T], respectively. Thus we obtain 


Rf + U t 

max(P, 1) J 


c 2 qE[I(j G PO/lPil] + Ci(q)c 2 q 2 /m 
c 2 qE[I(j G Pi)/|Pi|] + c 1 (q)c 2 q 2 /rn 
c i{q)q/m + c 2 qE[I(j G 1Zi)/\ 1Z\\] 
c 2 qE[I(j G Pi)/|Pi|] + ci(q)q/m 


if Hj — (0, 0), 
if Hj G {(1, 0), (—1, 0)}, 
if Hje {(o,i), (o,-i)}, 
if P, G {(1,-1), (-1,1)}, 


and for the directional error terms: 

?L 


E 


E 




lira x(P, 1) 

tdR 
max(P, 1) 


< 


< 


ci(q)q 

m 

for j with 

#i = (U) 

Ci(q)q 

m 

for j with 

Hi = (-1,-1) 


The result follows from using expression ([2]) for the directional FDR for replicability 
analysis, and summing up over the above upper bounds. 


B Proof of Theorem 14.11 


It is easy to show that the procedure in Section 14.21 is unchanged if we replace Step 
2 by the following: the replicability claims are all features with ff° n *(oi) < a. The 
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equivalence follows from the facts that ff ion (x) is a continuous function of x and 
ff onf (x)/x is strictly monotone decreasing (this result follows from the proof of 
Lemma Sl.l in the SI of [T| and it is straightforward to show that it continues to 
hold in the directional replicability analysis). 

We shall now prove that the expression in (00) with q replaced by a is an upper bound 
for the directional FWER for replicability analysis, which is Pr (R — S > 0). It was 
shown in the proof of Theorem 13.11 that this expression is at most a if /oo < /oo- Note 
that 

Pr (R-S > 0) < E{R-S ) < E E H) + E E{Rf) 

i,i)} {r-Hj={~ i-i)} 

+ E E ( R f + R i) 

{j-.Hjem-i), (0,1), (0,0), (1,0), (-1,0), (1,-1), (-1,1)}} 


We consider the equivalent procedure that replaces Step 2 by counting as replicability 
claims all features with ff on (a) < a (as discussed above). The directional error 
terms in the Erst two sums contribute the following: 

E ( r l\ < ci{a)a ^ for with = ^ ^ 

J rn 

E ( Rf ) < for j with Hj = (-1, -1) 

To see how these upper bounds were derived, we consider only the first (since the 
second is derived similarly). For j with Hj = (1,1) 

E (Rf) < Pr(P,j < min(ci (a)a/m, 0.5), P^j < c 2 a/Ri ) 

< Pr(Pf < ci{a)a/m) < Cia/m, 

where the hrst inequality follows from the fact that a replicability claim is made in 
the left direction only if Pf < Pfj, i.e., only if Pf < 0.5, and the last inequality 
follows that the fact that for H\ 3 = 1, Pf is stochastically larger than uniform. 

All remaining errors are false replicability claims, not only directional errors. Clearly, 

E(Rf + Rf) = Pr(min(P^, Pff) < d(a)a/m, P^ < c 2 a/\TZ 1 \,j £ IZi). 

It is simple to show (using similar derivations to these in the proof of Theorem S6.1 
in the SI of H) that the right hand side is at most the following upper bounds: 


E(Rf+Rf) < < 




c 2 aE[I(j £ TZi)/\TZi\] + c\(a)a/m x c 2 a 
c 2 aE[I(j £ 1 Zi)/\1Zi\] + ci(a)a/m x c 2 a 
ci(a)a/m + c 2 aE[I(j £ Pi)/|Pi|] 
c 2 aE[I(j £ TZi)/\lZi\\ + ci(a)a/m 


if Hj = (0,0), 
if Hj £ {(1,0), (-1,0)}, 
if Hj£ {(0,1), (0,-1)}, 
if ifj-G {(1,-1), (-1,1)} 


The result follows from summing over these upper bounds. 
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